Support group chats

[przemoc]

I use Google Messages with following configuration
Settings > Advanced > Group Messaging > Send an SMS reply to all recipients and get individual replies (mass text)

Such group chat (and corresponding individual messages) look like this in messages.ndjson exported in v2.3.1:

{"_id":"11038","thread_id":"155","address":"+48aaaaaaaaa +48bbbbbbbbb +48ccccccccc ddddddddd","date":"1695928662181","date_sent":"0","read":"1","status":"0","type":"2","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostru","locked":"0","error_code":"0","seen":"1","timed":"0","deleted":"0","sync_state":"0","marker":"0","bind_id":"0","mx_status":"0","out_time":"0","sim_id":"1","block_type":"0","advanced_seen":"3","b2c_ttl":"0","fake_cell_type":"0","url_risky_type":"0","favorite_date":"0","sub_id":"1","__display_name":"Person d"}
{"_id":"11041","thread_id":"3","address":"+48ccccccccc","date":"1695928662180","date_sent":"1695928665179","read":"1","status":"0","type":"2","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostru","locked":"0","error_code":"0","seen":"1","timed":"0","deleted":"0","sync_state":"0","marker":"0","bind_id":"0","mx_status":"0","out_time":"0","sim_id":"1","block_type":"0","advanced_seen":"3","b2c_ttl":"0","fake_cell_type":"0","url_risky_type":"0","favorite_date":"0","sub_id":"1","__display_name":"Person c"}
{"_id":"11040","thread_id":"11","address":"+48bbbbbbbbb","date":"1695928662180","date_sent":"1695928667041","read":"1","status":"0","type":"2","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostru","locked":"0","error_code":"0","seen":"1","timed":"0","deleted":"0","sync_state":"0","marker":"0","bind_id":"0","mx_status":"0","out_time":"0","sim_id":"1","block_type":"0","advanced_seen":"3","b2c_ttl":"0","fake_cell_type":"0","url_risky_type":"0","favorite_date":"0","sub_id":"1","__display_name":"Person b"}
{"_id":"11039","thread_id":"22","address":"+48aaaaaaaaa","date":"1695928662180","date_sent":"1695928665843","read":"1","status":"0","type":"2","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostru","locked":"0","error_code":"0","seen":"1","timed":"0","deleted":"0","sync_state":"0","marker":"0","bind_id":"0","mx_status":"0","out_time":"0","sim_id":"1","block_type":"0","advanced_seen":"3","b2c_ttl":"0","fake_cell_type":"0","url_risky_type":"0","favorite_date":"0","sub_id":"1","__display_name":"Person a"}
{"_id":"11042","thread_id":"41","address":"ddddddddd","date":"1695928662180","date_sent":"1695928665985","read":"1","status":"0","type":"2","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostru","locked":"0","error_code":"0","seen":"1","timed":"0","deleted":"0","sync_state":"0","marker":"0","bind_id":"0","mx_status":"0","out_time":"0","sim_id":"1","block_type":"0","advanced_seen":"3","b2c_ttl":"0","fake_cell_type":"0","url_risky_type":"0","favorite_date":"0","sub_id":"1","__display_name":"Person d"}

When imported, group chat doesn't look like a group chat, but like an individual chat with person whose name is concatenation of those phone numbers:

+48aaaaaaaaa+48bbbbbbbbb+48cccccccccddddddddd

(EDIT: To clarify, respective individual chats contain the message, as it is apparently stored for each of them individually too, so no issues with individual chats.)

I suspect that group chat may require some special handling on import, export or maybe in both cases.

---

Background:

Thanks for creating this application! I'm fan of SMS Backup+, but it doesn't handle MMSes properly, so when switching phones and restoring them, I were always losing them in respective conversation threads (at least they were still stored in mail most of the time, so it was somewhat acceptable). I hoped to avoid losing anything on target phone this time, so I gave your app a try.

I've used the latest SMS Import / Export, which is at v2.3.1 right now. First I exported messages from Android 11 phone (w/ Google Messages version messages.android_20240123_01_RC02.phone_dynamic). Then I imported messages to Android 14 phone (w/ Google Messages version messages.android_20240116_01_RC04.phone_dynamic). I'm mentioning Android (and Messages) versions just in case, not sure if they are relevant here at all. At first I thought it didn't work at all, because after finishing import, Messages didn't show anything, but after stopping the app, clearing its storage and running Messages again, messages were finally available! I've started scrolling to the bottom, and it seemed that chats from few years back were not showing, Messages displayed some placeholder at the bottom and some loading animations were happening. It felt like it got stuck, but eventually after several seconds, or maybe even a minute, more messages loaded, then I scrolled more and situation repeated a few times, until I reached to the very first message that was stored on the old phone. Hard to tell if everything imported properly (due to sheer volume, many thousands of messages spanning over 10 years), but quickly glancing at stuff looked like things looked good, and MMSes were in converation threads. But I noticed one problem.

[tmo1]

Thanks for the report; I'm looking into it.

[tmo1]

When imported, group chat doesn't look like a group chat, but like an individual chat with person whose name is concatenation of those phone numbers:

I have confirmed that the app does not handle SMS message with multiple recipients correctly.

I suspect that group chat may require some special handling on import, export or maybe in both cases.

It looks like it. Unfortunately, I have not yet been able to figure out how to do so.

after finishing import, Messages didn't show anything, but after stopping the app, clearing its storage and running Messages again, messages were finally available! I've started scrolling to the bottom, and it seemed that chats from few years back were not showing, Messages displayed some placeholder at the bottom and some loading animations were happening. It felt like it got stuck, but eventually after several seconds, or maybe even a minute, more messages loaded, then I scrolled more and situation repeated a few times, until I reached to the very first message that was stored on the old phone.

This is documented in the README.

[przemoc]

I have confirmed that the app does not handle SMS message with multiple recipients correctly.

It looks like it. Unfortunately, I have not yet been able to figure out how to do so.

Thank you for looking into that.

Would it be possible to add support for ignoring those group chats on import?
(I mean only group ones, individual ones should not be skipped, because same messages can be found in them too.)

This is documented in the README.

The behavior with Google Messages taking very long time to load older messages was not described. I was on the verge of thinking that something was broken after import. So decided to describe my experience, so that others trying to google it would be able to find it.

[tmo1]

Would it be possible to add support for ignoring those group chats on import?
(I mean only group ones, individual ones should not be skipped, because same messages can be found in them too.)

Can you explain more clearly what you mean by this?

[przemoc]

As I have shown in my first message in the issue, messages send through group chat (e.g. to +48aaaaaaaaa & +48bbbbbbbbb) appear also in individual chats (separately w/ +48aaaaaaaaa, and separately w/ +48bbbbbbbbb) when exported. Preserving group chats would be preferable for seamless transition between phones, but even if they're not imported, messages are not lost and still accessible (which is the most important feature of any backup & restore solution), and if one would like to send message to the same group chat that was previously created on other device, in the worst case it can be recreated manually for new messages.

So what I'm suggesting is having an option for import to ignore/skip group chats, i.e. entries with multiple numbers in address.
(And only for import, because export seems ok-ish as far as I can tell, so it's good to have group chats exported even if it cannot be imported properly atm.)
Having broken group chats and having to remove them manually is somewhat inconvenient.

When a way to properly import group chats will be found, then such option could be no longer needed (or it could be left there, maybe some folks would prefer it, or there could be even a counterpart option to simply import only group chats for those that had to skip it before).

Does it sound reasonable?

[tmo1]

It turns out that in my original testing of multiple recipient SMS messages, I was making a mistake and not creating them properly, and that's why I was not seeing what you were seeing. Now that I'm doing it correctly, I'm seeing results similar to yours, with the difference that for your sample message, I see what you describe, whereas with messages that I created, the message with the recipients concatenated is placed in the same thread as the message to the final recipien. I.e., a message to num1, num2, and num3 exports as four messages, as per your example, but upon import, I get three conversations containing four messages, with the message to "num1 num2 num3" combined in the same conversation as the one to num3. (I think the combination was always with the final recipient, although I'm not totally sure.)

In any event, after much experimenting and searching, I have still been unable to figure out how to properly import such messages, so I've followed your suggestion and modified the app to skip them. I'm leaving this issue open in the hope that we'll eventually figure out the correct way to handle such messages.

[przemoc]

Thank you for the effort. I will try to do new export + import in upcoming days and test if skipping works as expected.

[przemoc]

I tested v2.3.2 today. Group chats have been skipped during import, so there are no entries with broken recipient (seen as concatenated numbers previously). That's a good workaround until proper solution will be found.

[przemoc]

I recently tested importing again (before 2.4.0 was released) and noticed one problem, but was on the go and not able to share it earlier, so sorry for that.

Current workaround for group chats, which is simply skipping them on import:
https://github.com/tmo1/sms-ie/blob/92da6e66a879e3b9b1307629aac305b72da7bf49/app/src/main/java/com/github/tmo1/sms_ie/ImportExportMessages.kt#L391--L396
that was added in 2f07074 introduced regression.

Existing logic to detect group chats by having more than one word in address field is unfortunately insufficient, it leads to skipping some normal messages too.

You may have messages sent by phone services providers, government agencies, etc. that do not use numbers but names instead. For instance in Poland we receive occassionally messages from ALERT RCB (providing alerts from Government Centre for Security). Such messages are lost upon import now if there is a space in Sender ID.

I feel like detection for group chat messages needs to be more elaborate. Basically only messages where all words in address field are true phone numbers should be considered as such. If even one word is something else than phone number, then it should not be treated as group chat message.

I am not aware of how phone numbers are stored internally in Android across the world, but I would expect them to optionally start with plus sign (+) and have non-zero number of digits (0-9), which could be possibly written as following Java RegExp:
\A\+?\d+\Z (assuming matching would be done per word)
But maybe more non-digit characters are allowed beside plus sign, haven't researched that.

Alternatively one could try to simply detect characters other than plus sign, digits and space in address to bail out from group chat logic, but it feels like with this simpler approach some corner cases could be missed, like non number sender like +505+.

[chenxiaolong]

Basically only messages where all words in address field are true phone numbers should be considered as such.

It may be worth trying https://developer.android.com/reference/android/telephony/PhoneNumberUtils#formatNumber(java.lang.String,%20java.lang.String) to determine if a number is valid. (It returns null for invalid phone numbers.) I don't know how strict/lenient it is, but it might be easier than coming up with a regex.

---

As for the underlying issue, I wonder if it makes more sense to do something similar to Android's builtin TelephonyBackupAgent.

On export, for each thread, it grabs the Telephony.Threads.RECIPIENT_IDS, which is a space separated string of longs. It then does one call to query for phone numbers for all of these IDs. The list of recipients is stored as an array in their JSON export.

On import, it reads the JSON list of recipients and always imports them as a collection (vs. a string) using the Set<String> overload of Telephony.Threads.getOrCreateThreadId().

This same procedure is used for both SMS and MMS messages.

It still imports and exports Telephony.Sms.ADDRESS and Telephony.Mms.Addr.ADDRESS, but never parses them. Only Telephony.Threads.RECIPIENT_IDS is used to determine the recipients for threads.

[tmo1]

As for the underlying issue, I wonder if it makes more sense to do something similar to Android's builtin TelephonyBackupAgent.

No doubt it does - thanks for trawling the code to find this ...

On import, it reads the JSON list of recipients and always imports them as a collection (vs. a string) using the Set overload of Telephony.Threads.getOrCreateThreadId().

Hm. I was pretty sure that I had tried using the Set<String> form of Telephony.Threads.getOrCreateThreadId() to import group SMS message and that it hadn't worked, but I just tried it by passing in Telephony.Sms.ADDRESS.split(" ").toSet() from a group SMS message and it actually seems to work.

We still have the problem raised by @przemoc of distinguishing between address fields that contain multiple recipients and those that contain recipients that contain spaces. I suppose you're right that the ultimately correct solution is to store during export each message's recipients, obtained via the Telephony.Threads.RECIPIENT_IDS to addresses query you've found, and then to use those values upon import, but this would break backward compatibilty and so would have to be done very deliberately.

The real frustration here is that I look at messages as being primarily standalone objects, without any native concept of threads or conversations: IIUC, when Alice sends a message to Bob, that message as sent over the wire contains no information about any other, previous messages sent between them. Android, however, imposes its own notions of threads and conversations upon messages, and makes them an integral part of its message storage model. My app tries to untangle the messages from their Android-added structure, but inevitable runs into problems such as the ones in this issue.

Do I understand this correctly? Do you have a recommendation?

[chenxiaolong]

No problem!

I think ultimately, storing the array of addresses queried from RECIPIENT_IDS is probably the way to go. Compatibility with the current format (new app + old backup) could probably be done on a best effort basis, like with @przemoc's suggestion of splitting on whitespace and attempting to parse each as a phone number. I'm not sure about the old app + new backup scenario though.

The real frustration here is that I look at messages as being primarily standalone objects, without any native concept of threads or conversations: IIUC, when Alice sends a message to Bob, that message as sent over the wire contains no information about any other, previous messages sent between them. Android, however, imposes its own notions of threads and conversations upon messages, and makes them an integral part of its message storage model. My app tries to untangle the messages from their Android-added structure, but inevitable runs into problems such as the ones in this issue.

Yeah, this is indeed frustrating. I personally like the way sms-ie represents the messages. I'd imagine that since Android's built-in backups use a similar data model to yours, they wouldn't introduce anything that further complicates translation to and from Android's data model (hopefully...).