common.storages.load_package
ParsedLoadJobFileName Objects
class ParsedLoadJobFileName(NamedTuple)
Represents a file name of a job in load package. The file name contains name of a table, number of times the job was retired, extension and a 5 bytes random string to make job file name unique. The job id does not contain retry count and is immutable during loading of the data
job_id
def job_id() -> str
Unique identifier of the job
file_name
def file_name() -> str
A name of the file with the data to be loaded
with_retry
def with_retry() -> "ParsedLoadJobFileName"
Returns a job with increased retry count
PackageStorage Objects
class PackageStorage()
APPLIED_SCHEMA_UPDATES_FILE_NAME
updates applied to the destination
__init__
def __init__(storage: FileStorage, initial_state: TLoadPackageState) -> None
Creates storage that manages load packages with root at storage and initial package state initial_state
list_packages
def list_packages() -> Sequence[str]
Lists all load ids in storage, earliest first
NOTE: Load ids are sorted alphabetically. This class does not store package creation time separately.
list_failed_jobs_infos
def list_failed_jobs_infos(load_id: str) -> Sequence[LoadJobInfo]
List all failed jobs and associated error messages for a load package with load_id
import_job
def import_job(load_id: str,
job_file_path: str,
job_state: TJobState = "new_jobs") -> None
Adds new job by moving the job_file_path into new_jobs of package load_id
complete_loading_package
def complete_loading_package(load_id: str,
load_state: TLoadPackageState) -> str
Completes loading the package by writing marker file with`package_state. Returns path to the completed package
remove_completed_jobs
def remove_completed_jobs(load_id: str) -> None
Deletes completed jobs. If package has failed jobs, nothing gets deleted.
schema_name
def schema_name(load_id: str) -> str
Gets schema name associated with the package
get_load_package_info
def get_load_package_info(load_id: str) -> LoadPackageInfo
Gets information on normalized/completed package with given load_id, all jobs and their statuses.
is_package_partially_loaded
@staticmethod
def is_package_partially_loaded(package_info: LoadPackageInfo) -> bool
Checks if package is partially loaded - has jobs that are not new.