Commit 6fd433f2 authored by Marco Perronet's avatar Marco Perronet
Browse files

Change directory structure and filenames

parent 3d87ee68
Pipeline #57275 failed with stages
in 53 seconds
......@@ -33,4 +33,4 @@ testing/workloads/manual-input
*.csv
# Build output from trace-cmd's libraries
src/events_generation/binary_parser/lib/build_output
src/event_generation/binary_parser/lib/build_output
......@@ -23,7 +23,7 @@ stages:
build-libs:
stage: build
script:
- cd src/events_generation/binary_parser/lib
- cd src/event_generation/binary_parser/lib
- ./build_libs.sh
build:
......
[submodule "src/events_generation/binary_parser/lib/trace-cmd"]
path = src/events_generation/binary_parser/lib/trace-cmd
[submodule "src/event_generation/binary_parser/lib/trace-cmd"]
path = src/event_generation/binary_parser/lib/trace-cmd
url = https://github.com/cedric-courtaud/trace-cmd.git
[submodule "src/events_generation/binary_parser/lib/libtraceevent"]
path = src/events_generation/binary_parser/lib/libtraceevent
[submodule "src/event_generation/binary_parser/lib/libtraceevent"]
path = src/event_generation/binary_parser/lib/libtraceevent
url = https://github.com/perronet/libtraceevent.git
[submodule "src/events_generation/binary_parser/lib/libtracefs"]
path = src/events_generation/binary_parser/lib/libtracefs
[submodule "src/event_generation/binary_parser/lib/libtracefs"]
path = src/event_generation/binary_parser/lib/libtracefs
url = https://github.com/perronet/libtracefs.git
......@@ -65,21 +65,21 @@ Each module is implemented in the following files. The ones in bold contain the
### Events tracer
* [`events_generation/evg.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/events_generation/evg.rs): Generic definition of an event generator and of an event (TraceEvent).
* **[`events_generation/ftrace.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/events_generation/ftrace.rs)**: Event generator instantiated for `ftrace`. Reads from `trace_pipe` and parses low-level events in the form of a string to high-level TraceEvents used throughout RBFtrace.
* [`events_generation/dummy.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/events_generation/dummy.rs): Event generator instantiated for a synthetically generated trace.
* [`events_generation/replay.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/events_generation/replay.rs): Event generator instantiated for the replay of a trace.
* [`events_processing/invocation_cycle.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/events_processing/invocation_cycle.rs): Encodes the automaton to check for consistency of traced events. Processes events by adding extra information such as total execution time of an activation.
* **[`events_processing/evp.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/events_processing/evp.rs)**: Hot loop that processes one event at a time by using an instantiated event generator and the invocation cycle automaton. Sends periodic updates with new arrivals and RBFs to the model matcher.
* [`event_generation/evg.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/event_generation/evg.rs): Generic definition of an event generator and of an event (TraceEvent).
* **[`event_generation/ftrace.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/event_generation/ftrace.rs)**: Event generator instantiated for `ftrace`. Reads from `trace_pipe` and parses low-level events in the form of a string to high-level TraceEvents used throughout RBFtrace.
* [`event_generation/dummy.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/event_generation/dummy.rs): Event generator instantiated for a synthetically generated trace.
* [`event_generation/replay.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/event_generation/replay.rs): Event generator instantiated for the replay of a trace.
* [`event_processing/invocation_cycle.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/event_processing/invocation_cycle.rs): Encodes the automaton to check for consistency of traced events. Processes events by adding extra information such as total execution time of an activation.
* **[`event_processing/event_loop.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/event_processing/event_loop.rs)**: Hot loop that processes one event at a time by using an instantiated event generator and the invocation cycle automaton. Sends periodic updates with new arrivals and RBFs to the model matcher.
* [`real_time/arrival_sequence.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/arrival_sequence.rs): Contains the struct ArrivalSequence, which is sent as an update to the matcher thread.
* [`real_time/rbf_curve.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/rbf_curve.rs): Contains the struct RbfCurve along with the algorithm used to build RBF curves. RBFs are also sent as an update to the matcher thread.
### Configuration monitor
* [`feature_detection/features.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/feature_detection/features.rs): Defines all the system features that we want to detect in the struct SysConf.
* [`feature_detection/system.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/feature_detection/system.rs): Low-level functions to detect the system configuration (i.e. perform syscalls or read from the appropriate configuration files).
* [`feature_detection/multiproc_type.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/feature_detection/multiproc_type.rs): Functions to decide what the multiprocessor scheme is based on the affinity masks of traced threads.
* **[`feature_detection/detect.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/feature_detection/detect.rs)**: Main code that performs the actual detection and fills SysConf.
* [`config_detection/features.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/config_detection/features.rs): Defines all the system features that we want to detect in the struct SysConf.
* [`config_detection/system.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/config_detection/system.rs): Low-level functions to detect the system configuration (i.e. perform syscalls or read from the appropriate configuration files).
* [`config_detection/multiproc_type.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/config_detection/multiproc_type.rs): Functions to decide what the multiprocessor scheme is based on the affinity masks of traced threads.
* **[`config_detection/detect.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/config_detection/detect.rs)**: Main code that performs the actual detection and fills SysConf.
### Phase detector
......@@ -87,24 +87,24 @@ Currently not implemented. For now we only do the configuration detection as a p
### Model extractor
* [`real_time/analysis/models.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis/models.rs): Defines the models that we are currently supporting.
* **[`real_time/analysis/model_matching.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis/model_matching.rs)**: Performs model matching based on the new update received from the events tracer. Sends newly extracted models to the response-time analyzer.
* [`real_time/analysis_loopmodels.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis_loopmodels.rs): Defines the models that we are currently supporting.
* **[`real_time/analysis_loopmodel_matching.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis_loopmodel_matching.rs)**: Performs model matching based on the new update received from the events tracer. Sends newly extracted models to the response-time analyzer.
* [`real_time/arrival_sequence.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/arrival_sequence.rs): Contains the struct ArrivalSequenceSubset. This structure implements an algorithm that tries to keep only the most relevant observations rather than storing the whole trace in memory. The model matching is performed only on the observations stored in ArrivalSequenceSubset.
### Response-time analyzer
* **[`real_time/analysis/rta.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis/rta.rs)**: Contains the implementation of the response-time analyses that we are supporting.
* **[`real_time/analysis/analysis.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis/analysis.rs)**: Picks a response time-analysis to perform based on the extracted models and system configuration. Performs the analysis and reports the results.
* **[`real_time/analysis_looprta.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis_looprta.rs)**: Contains the implementation of the response-time analyses that we are supporting.
* **[`real_time/analysis_loopanalysis.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis_loopanalysis.rs)**: Picks a response time-analysis to perform based on the extracted models and system configuration. Performs the analysis and reports the results.
### Diagnostics generator
* [`real_time/analysis/analysis.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis/analysis.rs): If it wasn't possible to pick an analysis, prints why that is the case. (TODO we are not considering every case for now)
* [`real_time/analysis_loopanalysis.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis_loopanalysis.rs): If it wasn't possible to pick an analysis, prints why that is the case. (TODO we are not considering every case for now)
### Model exporter
* [`util/output.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/util/output.rs): Defines standard output format.
* [`util/export.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/util/export.rs): Creates directories and serializes to JSON.
* [`real_time/analysis/analysis.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis/analysis.rs): Writes the extracted models in JSON format at `testing/output/extracted_models` by using the export utils. (TODO `model_matching.rs` should do this instead)
* [`real_time/analysis_loopanalysis.rs`](https://gitlab.mpi-sws.org/perronet/rbf-trace/-/blob/master/src/real_time/analysis_loopanalysis.rs): Writes the extracted models in JSON format at `testing/output/extracted_models` by using the export utils. (TODO `model_matching.rs` should do this instead)
#### Other directories and files
......
......@@ -2,24 +2,24 @@ use std::process::Command;
fn main() {
// Tell Cargo that if the given file changes, to rerun this build script.
println!("cargo:rerun-if-changed=src/events_generation/binary_parser/lib/libtraceevent");
println!("cargo:rerun-if-changed=src/events_generation/binary_parser/lib/libtracefs");
println!("cargo:rerun-if-changed=src/events_generation/binary_parser/lib/trace-cmd");
println!("cargo:rerun-if-changed=src/event_generation/binary_parser/lib/libtraceevent");
println!("cargo:rerun-if-changed=src/event_generation/binary_parser/lib/libtracefs");
println!("cargo:rerun-if-changed=src/event_generation/binary_parser/lib/trace-cmd");
// Compile the trace-cmd libraries.
let mut build_libs_command = Command::new("./build_libs.sh");
build_libs_command.current_dir("src/events_generation/binary_parser/lib");
build_libs_command.current_dir("src/event_generation/binary_parser/lib");
build_libs_command.status().expect("Failed to build trace-cmd libraries.");
// Linker options for rustc (link trace-cmd libraries).
println!("cargo:rustc-link-lib=dylib=traceevent");
println!("cargo:rustc-link-lib=dylib=tracefs");
println!("cargo:rustc-link-lib=dylib=tracecmd");
println!("cargo:rustc-link-search=native=src/events_generation/binary_parser/lib/build_output/usr/lib64");
println!("cargo:rustc-link-search=native=src/event_generation/binary_parser/lib/build_output/usr/lib64");
// Without this we won't be able to find the library when running cargo run
// Note: this *only works* with cargo run
println!("cargo:rustc-env=LD_LIBRARY_PATH=src/events_generation/binary_parser/lib/build_output/usr/lib64");
println!("cargo:rustc-env=LD_LIBRARY_PATH=src/event_generation/binary_parser/lib/build_output/usr/lib64");
// Compile rtspin (for testing)
let mut build_rtspin_command = Command::new("make");
......@@ -27,5 +27,5 @@ fn main() {
build_rtspin_command.status().expect("Failed to build rtspin.");
// Generate bindings (reminder)
// bindgen -o src/bindings.rs src/events_generation/binary_parser/stream.h -- -I/usr/local/include/traceevent -I/usr/local/include/tracefs -I/usr/local/include/trace-cmd
// bindgen -o src/bindings.rs src/event_generation/binary_parser/stream.h -- -I/usr/local/include/traceevent -I/usr/local/include/tracefs -I/usr/local/include/trace-cmd
}
......@@ -13,7 +13,7 @@ pub struct Opt {
/// Run feature detection only
#[structopt(short = "d", long)]
pub feature_detection: bool,
pub config_detection: bool,
/// Turn off analysis
#[structopt(short = "a", long)]
......
use structopt::StructOpt;
use serde_yaml;
use std::fs;
fn main() {
}
use structopt::StructOpt;
use serde_yaml;
use std::fs;
use rbf_trace::util::helpers::*;
use rbf_trace::event_generation::evg::{
EventsGenerator,
TraceEvent,
};
fn main() {
// let args = Opt::from_args();
// let source: YamlSource;
// if is_json_file(&args.source_path) {
// source = YamlSource::new(&args.source_path);
// } else {
// // Binary
// }
// // TODO define tx_matcher
}
#[derive(Debug)]
struct YamlSource {
trace: Vec<TraceEvent>,
curr_event_idx: usize,
path: String,
}
impl YamlSource {
pub fn new(path: &String) -> Self {
let mut ret = YamlSource {
trace: Vec::new(),
curr_event_idx: 0,
path: String::new(),
};
ret.path = path.clone();
ret.setup();
ret
}
}
impl EventsGenerator for YamlSource {
fn setup(&mut self) {
self.trace = serde_yaml::from_str(&fs::read_to_string(&self.path).unwrap()).unwrap();
}
fn shutdown(&mut self) {
println!("Trace ended!");
}
fn next_event(&mut self) -> Option<TraceEvent> {
if self.curr_event_idx < self.trace.len() {
let ret: Option<TraceEvent>;
ret = Some(self.trace[self.curr_event_idx]);
self.curr_event_idx += 1;
return ret;
} else {
return None;
}
}
}
// #[derive(Debug)]
// struct BinarySource {
// rx: Receiver<EventUpdate>,
// }
// impl BinarySource {
// pub fn new(rx: Receiver<EventUpdate>) -> Self {
// BinarySource {
// rx: rx,
// }
// }
// }
// impl EventsGenerator for BinarySource {
// fn next_event() -> Option<TraceEvent>
// }
#[derive(Debug, StructOpt)]
pub struct Opt {
/// Specify the event source. Can be a JSON file or a pipe with binary data.
#[structopt(short = "s", long)]
pub source_path: String,
/// Specify the output. Can be a JSON file or a pipe with binary data.
#[structopt(short = "o", long)]
pub output_path: String,
// TUNABLES
/// Jitter bound
#[structopt(short = "J", long)]
pub jitter_bound: Option<Jitter>,
/// Complexity threshold
#[structopt(short = "C", long)]
pub complexity: Option<u128>,
/// Sparsify factor
#[structopt(short = "S", long)]
pub sparsify: Option<u64>,
/// Arrivals buffer size
#[structopt(short = "B", long)]
pub bufsize: Option<usize>,
/// Maximal busy window for all curves
#[structopt(short = "w", long)]
pub max_window: Option<usize>,
}
/* Helpers */
fn is_json_file(s: &String) -> bool {
s.ends_with(".json")
}
......@@ -6,7 +6,7 @@ use structopt::StructOpt;
use cpu_time::ThreadTime;
use rbf_trace::{sync::{EventUpdate, ModelUpdate},
feature_detection,
config_detection,
util::export,
args::Opt,
params,
......@@ -15,7 +15,7 @@ use rbf_trace::{sync::{EventUpdate, ModelUpdate},
};
fn main() {
let sys_conf: feature_detection::features::SysConf;
let sys_conf: config_detection::features::SysConf;
let wc_start = std::time::Instant::now(); // Wall-clock time
let runtime_start_main = ThreadTime::now(); // Time spent running
let mut runtime_tracer = std::time::Duration::new(0, 0);
......@@ -35,7 +35,7 @@ fn main() {
// Normal run
if params::replay_file().is_none() {
sys_conf = feature_detection::detect::detect_sys_conf();
sys_conf = config_detection::detect::detect_sys_conf();
if params::logger_replay() {
// Save sys conf to file to replay later
let run_name = export::generate_run_name(&sys_conf.target_pids);
......@@ -67,7 +67,7 @@ fn main() {
/*** Response-time analysis ***/
let analysis_handle = thread::spawn(move|| {
let runtime_start_analyzer = ThreadTime::now();
real_time::analysis::analysis::start_rta_thread(rx_analyzer, &sys_conf_);
real_time::analysis::analysis_loop::start_rta_thread(rx_analyzer, &sys_conf_);
let runtime_analyzer = runtime_start_analyzer.elapsed();
analyzer_runtime_tx.send(runtime_analyzer).expect("ANALYSIS: Send error");
});
......@@ -75,7 +75,7 @@ fn main() {
/*** Model matching ***/
let matching_handle = thread::spawn(move|| {
let runtime_start_matcher = ThreadTime::now();
real_time::analysis::model_matching::start_matching_thread(rx_matcher, tx_matcher);
real_time::model_matching::matching_loop::start_matching_thread(rx_matcher, tx_matcher);
let runtime_matcher = runtime_start_matcher.elapsed();
matcher_runtime_tx.send(runtime_matcher).expect("MATCHING: Send error");
analysis_handle.join().unwrap();
......
......@@ -8,12 +8,12 @@ use std::path::PathBuf;
use std::io::Write;
use rbf_trace::util::helpers::*;
use rbf_trace::events_generation::evg::{
use rbf_trace::event_generation::evg::{
EventsGenerator,
};
use rbf_trace::events_generation::ftrace::FTraceEVG;
use rbf_trace::feature_detection::system::get_pids_with_policy;
use rbf_trace::feature_detection::features::SchedPolicy;
use rbf_trace::event_generation::ftrace::FTraceEVG;
use rbf_trace::config_detection::system::get_pids_with_policy;
use rbf_trace::config_detection::features::SchedPolicy;
fn main() {
let args = Opt::from_args();
......
use std::io::prelude::*;
use crate::feature_detection::system::*;
use crate::feature_detection::features::*;
use crate::feature_detection::multiproc_type::*;
use crate::config_detection::system::*;
use crate::config_detection::features::*;
use crate::config_detection::multiproc_type::*;
use crate::util::helpers::*;
pub fn detect_sys_conf() -> SysConf {
......
......@@ -2,7 +2,7 @@ use crate::util::helpers::*;
use std::fs::*;
use std::collections::HashMap;
use serde::{Serialize, Deserialize};
use crate::feature_detection::system::*;
use crate::config_detection::system::*;
#[derive(Debug, Copy, Clone, PartialEq, Serialize, Deserialize)]
pub enum SchedPolicy {
......
use std::collections::HashSet;
use crate::feature_detection::features::*;
use crate::config_detection::features::*;
use crate::util::helpers::*;
use crate::feature_detection::system::*;
use crate::config_detection::system::*;
use std::iter::FromIterator;
pub fn get_multiproc_type() -> MultiprocType {
......
......@@ -3,7 +3,7 @@ use nc;
use std::fs::*;
use vec_tree::VecTree;
use crate::feature_detection::features::*;
use crate::config_detection::features::*;
use crate::util::helpers::*;
const CGROUP_PATH_CPU: &str = "/sys/fs/cgroup/cpu"; // Cpu controller of cgroup v1
......@@ -138,10 +138,10 @@ pub fn set_clusters(sys_conf: &mut SysConf) {
}
},
MultiprocType::CLUSTERED => {
sys_conf.rt_threads_info_clusters = crate::feature_detection::multiproc_type::check_clustered(&sys_conf.rt_pids, true).unwrap();
sys_conf.rt_threads_info_clusters = crate::config_detection::multiproc_type::check_clustered(&sys_conf.rt_pids, true).unwrap();
},
MultiprocType::CLUSTEREDNF => {
sys_conf.rt_threads_info_clusters = crate::feature_detection::multiproc_type::check_clustered(&sys_conf.rt_pids, false).unwrap();
sys_conf.rt_threads_info_clusters = crate::config_detection::multiproc_type::check_clustered(&sys_conf.rt_pids, false).unwrap();
},
_ => { /* No clusters */ }
}
......
// use crate::util::helpers::*;
// use crate::events_generation::evg::*;
// use crate::event_generation::evg::*;
// /* Generate a period constant pattern ARPRD, with only one process */
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment