I am also a software engineer who turned data scientist. I love what I do and I am so glad I made the switch. But it is true that is very tuff (especially because it is a new profession yet to be fully understand and defined). Among every thing you said, the biggest truth is the one about being lonely. You are generaly alone working on your problem. This one made me laugh big time:

“Be prepared to have that conversation with yourself… or a rubber duck.”

I do miss the conversations and morning stand-ups that I had with my former software engineer colleagues…