Debugging DataSelector

[?]

Jul 9, 2021, 8:45 AM

A6ZAYJNBYFXPUTCHTQDGAWUZIYOXNO3IJBN73ZX6PEJ3T4NMNOGQC

Dependencies

[2] SAHJYVNB Removed checking functionality.
[3] AT753JPO Selected US unemployment series.
[4] GQVS55HI Finished generate_ts_spec() function.
[5] AIFRDCG2 Split off countries mod into a separate crate.
[6] U4VCAFXQ Added data_type to TSSpec key.
[7] 2SABVMY3 Finished into_json() functionality.
[8] TTR5IFSG Working on building generic TSSpec.
[9] 77SIQZ3E Separating out spec generation.
[10] XI5ALEH6 Take advantage of keytree FromStr functionality.
[11] LVMGQJGH Finished framework for checking series specifications with data.
[12] CUADTSHQ Save csv data as multiple files.
[13] JTX5OHWH Added USA CPI.
[14] GUXZCEWW Added Country enum.
[15] 4MG5JFXT First record.
[16] 5POF332L Working on fn cpi_included().

Change contents

file deletion: check_data.rs (-xw-xw-x--)

[3.6]→[3.13211:13226](∅→∅),[3.13226]→[3.13227:13227](∅→∅)

use std::collections::{
BTreeMap,
HashMap,
};
use std::convert::TryInto;
use std::fs;

use serde::Serialize;
use time::OffsetDateTime;

use countries::Country;
use fred_api::{
Fred,
};
use keytree::{KeyTree, KeyTreeRef};
use keytree::serialize::{
IntoKeyTree,
KeyTreeString,
};
use time_series::{
RegularTimeSeries,
TimeSeries,
};

use crate::serve_ts::{
TSGraphicSpec,
TSPageSpec,
};

impl DataSpec {
/// Check for errors that might have occurred somewhere along the toolchain.
let mut builder = CheckedDataSpec(Vec::new());

}
builder
// This is a specification for CheckedData
/// ```
/// series:
/// data_type: u
/// country: United States
/// id: LRUNTTTTUSQ156S
/// ```

impl<'a> TryInto<CheckedDataSpec> for KeyTreeRef<'a> {

Ok(
)
}
}

impl IntoKeyTree for CheckedDataSpec {
fn keytree(&self) -> KeyTreeString {
let mut kt = KeyTreeString::new();
for series in &self.0 {
kt.push_keytree(1, series.keytree());
}
kt
}
}

/// `CheckedSourceSeries` is a code representation of the 'checked_data.keytree' specification file.
/// It acts as the gate-keeper to the data on disk.
pub struct CheckedSourceSeries {
impl IntoKeyTree for CheckedSourceSeries {
fn keytree(&self) -> KeyTreeString {
let mut kt = KeyTreeString::new();
kt.push_key(0, "series");
kt
}
}

impl<'a> TryInto<CheckedSourceSeries> for KeyTreeRef<'a> {
type Error = keytree::Error;

fn try_into(self) -> Result<CheckedSourceSeries, keytree::Error> {

Ok(
CheckedSourceSeries{
}
)
}
}

data_type: self.value("series::data_type")?,
country: self.value("series::country")?,
series_id: self.value("series::series_id")?,
drop_first: self.opt_value("series::drop_first")?,
let country: Country = self.value("series::country")?;
// let time_stamp = match date_str {
// Some(date_str) => {
// Some(OffsetDateTime::parse(date_str, "%Y %T").unwrap())
// },
// None => None,
// };
kt.push_value(1, "data_type", &self.data_type.to_string());
kt.push_value(1, "country", &self.country.to_string());
kt.push_value(1, "series_id", &self.series_id.to_string());
kt.push_opt_value(1, "drop_first", self.drop_first);
pub data_type: DataType,
///
pub country: Country,
///
///
}

pub drop_first: Option<usize>,
pub series_id: SeriesId,

///
kt.push_key(0, "page");
CheckedDataSpec(self.vec_at("seriess::series")?)
fn try_into(self) -> Result<CheckedDataSpec, Self::Error> {
type Error = keytree::Error;
pub struct CheckedDataSpec(pub Vec<CheckedSourceSeries>);
}
}

for series_spec in &self.0 {
let checked_series = CheckedSourceSeries {
data_type: series_spec.data_type,
country: series_spec.country,
series_id: series_spec.series_id.clone(),
drop_first: None,
};
builder.0.push(checked_series);
pub fn into_checked(&self) -> CheckedDataSpec {
TSSeriesSpec,
TSSpec,
PageKey,
use crate::build_spec::{
DataSpec,
};
use crate::error::*;
use crate::{
DataType,
SeriesId,
};
use std::str::FromStr;
replacement in src/main.rs at line 1

[3.10]→[3.8660:8700](∅→∅),[3.8700]→[3.11575:11576](∅→∅),[3.12506]→[3.11575:11576](∅→∅),[3.11575]→[3.11575:11576](∅→∅),[3.11576]→[3.8701:8725](∅→∅)

use std::convert::TryInto;
use std::fs;

use countries::Country;

[3.10]

[3.8725]

use countries::*;
edit in src/main.rs at line 5

[3.1]

[3.8783]

use countries::Country;
use ui_data::DataType;
edit in src/main.rs at line 11

[2.1657]

[3.110]

use ui_data::fred::*;
replacement in src/main.rs at line 16

[3.9637]→[2.1658:1753](∅→∅)

let spec = DataSpec::from_file("source_data.keytree");
println!("{}", spec.keytree());

[3.9637]

[2.1753]

println!("{}", DataSelector::from_file("series_selector.keytree")
.resume_into_data_spec(Country::UnitedKingdom, DataType::Inf)
.unwrap()
.keytree());
replacement in src/lib.rs at line 6

[3.10274]→[3.9352:9414](∅→∅)

//! ## Step 1. Find the data of Fred and build a spec for it.

[3.10274]

[3.11914]

//! ## Step 1. Build a generic data specification.
replacement in src/lib.rs at line 8

[3.11918]→[3.11918:12035](∅→∅),[3.12035]→[3.10391:10572](∅→∅),[3.10391]→[3.10391:10572](∅→∅),[3.10572]→[3.12036:12116](∅→∅)

//! Probably the most effective way to find stuff, is to search on tags and then filter on series
//! title, such as
//! ```
//! for item in Fred::tags_series("usa;interest")
//! .unwrap()
//! .series()
//! .has_phrase("Prime")
//! .iter()
//! {
//! println!("{}", item);
//! }
//! ```
//! The filters such as `has_phrase` are found in the `fred_api` crate.

[3.11918]

[3.12116]

//! The data series selector specification looks like
replacement in src/lib.rs at line 10

[3.12120]→[3.9415:9466](∅→∅),[3.9466]→[3.12174:12570](∅→∅),[3.12844]→[3.12174:12570](∅→∅),[3.12174]→[3.12174:12570](∅→∅),[3.12570]→[2.1757:1810](∅→∅)

//! ## Step 2. Build a generic data specification.
//!
//! The results of Step 1 are integrated into algorithms in the `build_spec` module. These can then
//! be used to generate collections of series.
//!
//! These specifications are then edited by hand, and cannot be programatically regenerated after
//! editting. To modify the spec. programmatically, a newly generated `Keytree` string can be
//! manually appended.
//!
//! Example:
//! ```
//! println!("{}", generic_data_spec().keytree());

[3.12120]

[3.12621]

//! ```text
//! selectors:
//! series:
//! country: Australia,
//! data_type: u
//! tag: unemployment
//! none_of: Male
//! none_of: Female
//! none_of: 55-64
//! none_of: 25-54
//! none_of: 15-24
//! none_of: 20 to 24
//! none_of: Youth
//! none_of: Women
//! none_of: Teenagers
//! not_any_of: Rate
//!
//! series:
//! country: Austria,
//! data_type: u
//! tag: unemployment
//! none_of: Male
//! none_of: Female
//! none_of: 55-64
//! none_of: 25-54
//! none_of: 15-24
replacement in src/lib.rs at line 37

[3.12629]→[3.12629:12685](∅→∅)

//! Save the output as file `source_data.keytree`. Then

[3.12629]

[3.10572]

//! To take the data series selector specification and output a data series specification, use
//!
replacement in src/lib.rs at line 40

[3.10580]→[3.12686:12781](∅→∅)

//! let spec = DataSpec::from_file("source_data.keytree");
//! println!("{}", spec.keytree());

[3.10580]

[3.4836]

//! println!("{}", DataSpec::from_selector(`series_selector.keytree`).unwrap().keytree());
replacement in src/lib.rs at line 42

[3.4844]→[3.4844:4892](∅→∅)

//! The specification will look something like,

[3.4844]

[3.12781]

//!
//! Copy the output from the terminal and paste the results in `source_data.keytree`.
//! The data series specification `source_data.keytree` will look something like
//!
replacement in src/lib.rs at line 70

[3.12793]→[2.1811:1876](∅→∅)

//! ## Step 3. Use the data specification to write data to file.

[3.12793]

[3.9648]

//! ## Step 2. Use the data specification to write data to file.
replacement in src/lib.rs at line 72

[3.9656]→[2.1877:1939](∅→∅)

//! let checked = DataSpec::from_file("source_data.keytree");

[3.9656]

[2.1939]

//! let source = DataSpec::from_file("source_data.keytree");
replacement in src/lib.rs at line 82

[3.10171]→[3.10171:10241](∅→∅)

//! ## Step 5. Generate a generic time-series graphics specification.

[3.10171]

[3.10241]

//! ## Step 3. Generate a generic time-series graphics specification.
replacement in src/lib.rs at line 84

[3.10245]→[3.10245:10359](∅→∅)

//! This is used to generate time-series plots. Once it is generated in full, then it can be edited
//! manually.

[3.10245]

[3.10359]

//! The generated specification specifies the time-series graphics.
replacement in src/lib.rs at line 91

[3.6746]→[3.10406:10439](∅→∅)

//! ## Step 6. Serve Time-series

[3.6746]

[3.6780]

//! ## Step 4. Serve Time-series
replacement in src/lib.rs at line 93

[3.6784]→[3.6784:7089](∅→∅)

//! The structure of JSON which is served to the client is determined by `Serialize`-able data
//! structures. The top structure is `TSJson`. The `ts_spec.keytree` file specifies the graphics on
//! each page, and what data each graphic uses. The `ts_spec.keytree` file is deserialized into
//! `TSSpec`.

[3.6784]

[3.13017]

//! This step converts the specification in JSON data and meta-data that is served to the client.
//! The client Javascript then builds the graphics dynamically. Take both specifications and load
//! data from "/full/path/to/data".
edit in src/lib.rs at line 97

[3.13021]→[3.10440:10522](∅→∅),[3.10522]→[3.7191:7297](∅→∅),[3.7191]→[3.7191:7297](∅→∅),[3.7297]→[3.10523:10528](∅→∅)

//! The procedure is generally to read in the specification in `ts_spec.keytree`.
//! `data_spec.keytree` which specifies data available, and also acts as a gate-keeper for that
//! data.
//!
replacement in src/lib.rs at line 98

[3.7305]→[3.10529:10616](∅→∅)

//! let data_spec = CheckedDataSpec::from_file("checked_data.keytree").into_indexed();

[3.7305]

[3.7469]

//! let data_spec = DataSpec::from_file("checked_data.keytree").into_indexed();
replacement in src/lib.rs at line 102

[3.10629]→[3.10629:10699](∅→∅)

//! Take both specifications and load data from "/full/path/to/data".

[3.10629]

[3.7525]

//! Then `TSSpec` uses `IndexedDataSpec` as an argument to be converted into `TSJson`.
replacement in src/lib.rs at line 107

[3.7695]→[3.7695:7797](∅→∅),[3.7797]→[3.12953:13168](∅→∅),[3.13021]→[3.12953:13168](∅→∅)

//! Then `TSSpec` uses `IndexedCheckedDataSpec` as an argument to be converted into `TSJson`.
//!
//!
//! We want to serve data to clients in a particular way depending on the client (time-series or
//! scatter plots). Each client-facing data-structure has its own specification. The two
//! specifications interact.

[3.7695]

[3.13090]

//! All JSON is generated at server startup into an immutable `HashMap` in memory.
replacement in src/lib.rs at line 109

[3.13094]→[3.10779:10816](∅→∅)

//! ## Step 7. Serve UI Scatterplots

[3.13094]

[3.13133]

//! ## Step 5. Serve UI Scatterplots
replacement in src/lib.rs at line 424

[2.7399]→[2.7399:7464](∅→∅)

let v: Vec<SeriesSpec> = self.vec_at("series::series")?;

[2.7399]

[2.7464]

let v: Vec<SeriesSpec> = self.vec_at("seriess::series")?;