transientNamer.astronotes module

Tools to download, parse and ingest astronote content into a MySQL database

Author

David Young

Date Created

January 15, 2021

class astronotes(log, dbConn=False, settings=False)

Bases: object

Tools to download, parse and ingest astronote content into a MySQL database

Key Arguments:

• log – logger

• dbConn – database connection. Default False

• settings – the settings dictionary

Usage:

To setup your logger, settings and database connections, please use the fundamentals package (see tutorial here).

To initiate a astronotes object, use the following:

from transientNamer import astronotes
an = astronotes(
    log=log,
    dbConn=dbConn,
    settings=settings
)

download(cache_dir, inLastDays=False)

Download astronotes reported in the lasy N days. Check cache for notes alreaedy downloaded.

Key Arguments:

- `cache_dir` -- the directory to cache the json notes to.
- `inLastDays` -- download only notes reported in the last N days. Default *False*. (Download all)

Return:

- `downloadCount` -- number of new files cached

Usage:

from transientNamer import astronotes
an = astronotes(
    log=log,
    dbConn=dbConn,
    settings=settings
)
downloadCount = an.download(
    cache_dir=settings["astronote-cache"], inLastDays=30)
print(f"{downloadCount} new astronotes downloaded anc cached")

get_all_noteids(inLastDays=False)

get the noteids of those notes released in the last N days

Key Arguments:

• inLastDays – report only notesIds released in the last N days. Default False. (Report all)

Return:

- `noteIds` -- list of all reported noteIds

Usage:

from transientNamer import astronotes
an = astronotes(
    log=log,
    settings=settings
)
noteIds = an.get_all_noteid(inLastDays=3000)
print(f"Astronote IDs: {noteIds}")

notes_to_database()

read the notes and import them into indexed MySQL database tables

Usage:

from transientNamer import astronotes
an = astronotes(
    log=log,
    dbConn=dbConn,
    settings=settings
)
an.notes_to_database()

_create_db_tables()

create the astronote database tables if they don’t yet exist

Usage:

from transientNamer import astronotes
an = astronotes(
    log=log,
    dbConn=dbConn,
    settings=settings
)
an._create_db_tables()

_parse_json_to_database(skipAstronoteIds)

parse the cached json files and add content to mysql database tables

Key Arguments:

- `skipAstronoteIds` -- the astronote IDs already present in the database - do not reparse

_parse_html_to_database(skipAstronoteIds)

parse the HTML versions of the astronotes to database tables

Key Arguments:

- `skipAstronoteIds` -- the astronote IDs already present in the database - do not reparse