support the new json schema

[dmd]

In the new on-device-only Location History, Google has changed the JSON schema.

[ryangriggs]

Does the app fail to work with the new schema?

[dmd]

Yes, it refuses to even load it.

"Invalid data: must contain 'timelineObjects' array."

[dmd]

(I'm a little surprised that you wrote this just last week, given Google Timeline is being sunsetted literally as we speak in favor of the new on-device version - were you unaware?)

[ryangriggs]

Wrote the app a few months ago, another user pitched in last week to add features.

How did you obtain your location history data? Did you use Takeout or another method?

[dmd]

No, Takeout is no longer supported for Location History. You have to export it from the app on your phone now.

https://support.google.com/maps/answer/14169818?co=GENIE.Platform%3DDesktop&oco=1

https://www.androidauthority.com/google-maps-killing-timeline-web-access-3449017/

https://gizmodo.com/google-maps-timeline-app-browser-1851520501

[dmd]

The new schema appears to be:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Generated schema for Root",
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "endTime": {
        "type": "string"
      },
      "startTime": {
        "type": "string"
      },
      "visit": {
        "type": "object",
        "properties": {
          "hierarchyLevel": {
            "type": "string"
          },
          "topCandidate": {
            "type": "object",
            "properties": {
              "probability": {
                "type": "string"
              },
              "semanticType": {
                "type": "string"
              },
              "placeID": {
                "type": "string"
              },
              "placeLocation": {
                "type": "string"
              }
            },
            "required": [
              "probability",
              "semanticType",
              "placeID",
              "placeLocation"
            ]
          },
          "probability": {
            "type": "string"
          }
        },
        "required": [
          "hierarchyLevel",
          "topCandidate",
          "probability"
        ]
      },
      "activity": {
        "type": "object",
        "properties": {
          "probability": {
            "type": "string"
          },
          "end": {
            "type": "string"
          },
          "topCandidate": {
            "type": "object",
            "properties": {
              "type": {
                "type": "string"
              },
              "probability": {
                "type": "string"
              }
            },
            "required": [
              "type",
              "probability"
            ]
          },
          "distanceMeters": {
            "type": "string"
          },
          "start": {
            "type": "string"
          }
        },
        "required": [
          "end",
          "topCandidate",
          "distanceMeters",
          "start"
        ]
      },
      "timelinePath": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "point": {
              "type": "string"
            },
            "durationMinutesOffsetFromStartTime": {
              "type": "string"
            }
          },
          "required": [
            "point",
            "durationMinutesOffsetFromStartTime"
          ]
        }
      },
      "timelineMemory": {
        "type": "object",
        "properties": {
          "destinations": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "identifier": {
                  "type": "string"
                }
              },
              "required": [
                "identifier"
              ]
            }
          },
          "distanceFromOriginKms": {
            "type": "string"
          }
        },
        "required": [
          "distanceFromOriginKms"
        ]
      }
    },
    "required": [
      "endTime",
      "startTime"
    ]
  }
}

which is to say, total garbage - it's an array of lots of different kinds of objects.

[ryangriggs]

Odd, I don't have that feature on my device yet... it still takes me directly to Takeout to download the location history. I guess they haven't rolled out the update to my account yet.

Do you have any links describing the schema of the new export format? A cursory search yielded nothing useful. I don't have much time today to look into this.

[dmd]

Also, just emailed you.

[dmd]

I dug into this a bit more. It's not as bad as all that. There are four types of objects - visit, activity, timelineMemory, and timelinePath. As best as I can tell given my own data - and I have continuous data since 2010, so it should be pretty representative - the first three of those can be ignored if all you're interested in is a list of places you've been.

I wrote this:

import json
import csv
import argparse
from datetime import datetime, timedelta


def parse_iso8601(timestamp):
    return datetime.fromisoformat(timestamp.replace("Z", "+00:00"))


def extract_lat_lon(location):
    return location.split(":")[1].split(",")


def main(input_file, output_file):
    with open(input_file, "r") as f:
        data = json.load(f)

    with open(output_file, "w", newline="") as csvfile:
        fieldnames = ["timestamp", "latitude", "longitude"]
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()

        for item in data:
            if "timelinePath" in item:
                path_points = item["timelinePath"]
                start_time = parse_iso8601(item["startTime"])
                for point in path_points:
                    offset = timedelta(
                        minutes=int(point["durationMinutesOffsetFromStartTime"])
                    )
                    timestamp = (start_time + offset).isoformat()
                    lat, lon = extract_lat_lon(point["point"])
                    writer.writerow(
                        {"timestamp": timestamp, "latitude": lat, "longitude": lon}
                    )


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Convert JSON to CSV.")
    parser.add_argument("input_file", help="Input JSON file")
    parser.add_argument("output_file", help="Output CSV file")
    args = parser.parse_args()

    main(args.input_file, args.output_file)

which appears to correctly parse the data into something that e.g. kepler.gl can read. Basically, for each object in the top level JSON array, process it only if it is a timelinePath object (it contains that key). Get the startTime, then for each item in the timelinePath array, take its latitude and longitude, and pair it with the startTime + the durationMinutesOffsetFromStartTime.

I tried doing it that way vs. also processing visits and activities and ended up with almost identical results, so probably this is the way to go. (Note that if you do go the way of also processing visits and activities, you must ignore visits that have a semanticType of "Searched Address".)

timelineMemory can always be ignored.

[dmd]

I should also say that, unfortunately, the exported data from the app is incomplete. It's pretty good but it's only about 20% of the data points. For example:

versus

[ryangriggs]

Hi Dr. Drucker, thanks for this info. Are you interested in adding a PR to support the new location data format? Not sure what you mean by exported data is incomplete. Do you mean my app doesn't export all the locations that you import from the file? Thanks for your input on this issue.

…

------------------------- Ryan Griggs Hilltop Computing www.hilltop.net 859-328-3223 Toll Free: 1 (888) 5-HILLTOP (888-544-5586)

On Wed, Jun 12, 2024 at 10:39 AM Daniel M. Drucker, Ph.D. < ***@***.***> wrote: I should also say that, unfortunately, the exported data from the app is incomplete. It's *pretty* good but it's only about 20% of the data points. For example: image.png (view on web) <https://github.com/ryangriggs/GoogleTimelineMapper/assets/41439/b5573670-d076-4d3a-a4b7-c7048521ea51> versus image.png (view on web) <https://github.com/ryangriggs/GoogleTimelineMapper/assets/41439/8a720527-e9fc-4a86-84b2-96d4dcfdee15> — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMVKGLVUKRFMFLHKSO4QMDZHBMR7AVCNFSM6AAAAABJGIH2ZSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRTGE4DQNRTGU> . You are receiving this because you commented.Message ID: ***@***.***>

[dmd]

Yeah, gimme a day and I'll PR.

No, that's a complaint about Google, not you. The export from the iOS google maps app doesn't contain the same amount of detail as the old Google Takeout data.

[dmd]

Actually, I don't know how you'd want this to work. Your app wants arrived, departed, duration, lat, lon. Using the timelinePath objects (which appear to contain the bulk of the data) we only really get a timestamp, lat, and lon.

[MueJosh]

Actually, I don't know how you'd want this to work. Your app wants arrived, departed, duration, lat, lon. Using the timelinePath objects (which appear to contain the bulk of the data) we only really get a timestamp, lat, and lon.

I can have a look at it in a few days, for now I'm working on a different feature (import GPX and kml and some other stuff).

[assc1967]

Hi to all,
I was trying to use your script for a project but like some other people found before the json downloaded from the device is probably different from the schema your script seems to be expecting
Maybe someone could give me some hint about how to find the problem and get the script working back? :)
Thank you