neurobooth_terra.dataflow.delete_files

neurobooth_terra.dataflow.delete_files(db_table, target_dir, suitable_dest_dir, threshold=0.9, older_than=30)

Delete files if x% of disk is filled.

Parameters

db_tableinstance of Table

The database table containing information about file transfers. The row corresponding to the transferred file is updated with is_deleted=True.

target_dirstr

The source directory path from which to delete files.

suitable_dest_dirstr

The destination directory path where the files should have been copied.

thresholdfloat

A fraction between 0. and 1. If more than threshold fraction of the source directory is filled, these files will be deleted.

older_thanint

If a file is older than older_than days in both src_dir and suitable_dest_dir, they will be deleted.

Notes

Let’s say the data was first written to “NAS”, then copied to “who” and from “who” to “neo”:

src	dest	is_deleted
NULL	NAS	False
Nas	who	False
who	neo	False

Now, if we want to delete the file from “Nas” and ensure that it has been copied to the final destination “neo”, we provide the following arguments to the function:

target_dir = NAS, suitable_dest = neo

src	dest	is_deleted
NULL	NAS	True
NAS	who	False
who	neo	False

Note that is_deleted bool is set to True where dest = target_dir

Here is another example:

target_dir = who, suitable_dest = neo

src	dest	is_deleted
NULL	NAS	True
NAS	who	True
who	neo	False