I am also a software engineer who turned data scientist. I love what I do and I am so glad I made the switch. But it is true that it is very hard (especially because it is a new profession yet to be fully understood and defined). Among everything you said, the biggest truth is the one about being lonely. You are generally alone working on your problem. This one made me laugh big time:

“Be prepared to have that conversation with yourself… or a rubber duck.”

I do miss the conversations and morning stand-ups that I had with my former software engineer colleagues…