Clustering network data

Clustering with infomap algorithm

class semanticlayertools.clustering.infomap.Clustering(inpath: str, outpath: str, recreate: bool = False, silent: bool = True, num_trials: int = 5, flow_model: str = 'undirected', debug: bool = False)

Cluster mulitlayer time-dependent networks using the infomap algorithm.

Calculates clusters using the infomap algorithm. Input files are assumed to have multilayer Pajek format and contain the year in four digit format. The default settings for running the method assume an undirected multilayer network and will use at most 5 optimization runs.

Parameters:

inpath (str) – Path to input pajek files
outpath (str) – Path for writing resulting cluster data
recreate (bool) – Toggle recreation of already exisiting files
silent (bool) – Toogle verbose mode for infomap
num_trials (int) – Number of runs for the infomap routine
flow_model (str) – Model for flow, directed or undirected.
debug (bool) – Toggle writing of debug info to standard output.

See also

Martin Rosvall and Carl T. Bergstrom (2008). Maps of information flow reveal community structure in complex networks. PNAS, 105, 1118. 10.1073/pnas.0706851105

calcInfomap(inFilePath, writeStates=False, depthLevel=1)

Calculate clusters for one pajek file.

Writes found cluster (i.e. module) information in CLU and FlowTree file format to output path.

Parameters:: inFilePath (str) – Path to input pajek file
Raises:: OSError – If one of the output files for this year already exists.
Returns:: Writes two files with found cluster information, method return value is empty
Return type:: None

See also

Infomap python documentation on mapequation Infomap module

run(states=False, depth=1): Calculate infomap clustering for all pajek files in input path.

Clustering using Leiden algorithm

Find temporal clusters in multilayer networks.

class semanticlayertools.clustering.leiden.TimeCluster(inpath: Path, outpath: Path, timerange: tuple = (1945, 2005), *, useGC: bool = True, debug: bool = False)

Cluster time-sliced data with the Leiden algorithm.

Calculates temporal clusters of e.g. time-sliced cocitation or citation data, using the Leiden algorithm . Two nodes are assumed to be identical in different year slices, if the node name is the same. This could be e.g. the bibcode or DOI.

Input files are assumed to include the year in the filename, have an ending _GC.net to denote their giant component character and should be in Pajek format. Alternatively, NCOL format is supported with the ending .ncol for general network data in column format.

The resolution parameter can be seen as a limiting density above which neighbouring nodes are considered a cluster. The interslice coupling describes the influcence of yearly order on the clustering process. See doc for the Leiden algorithm for more detailed info.

Parameters:

inpath (str) – Path for input network data
outpath (str) – Path for writing output data
resolution (float) – Main parameter for the clustering quality function (Constant Pots Model)
intersliceCoupling (float) – Coupling parameter between two year slices, also influences cluster detection
timerange (tuple) – The time range for considering input data (default=1945,2005))
useGC (bool) – If True use giant component for input data (format Pajek), if False use full network data in NCOL format.

Raises:

OSError – If the output file already exists at class instantiation

See also

Traag, V.A., Waltman. L., Van Eck, N.-J. (2018). From Louvain to Leiden: guaranteeing well-connected communities. Scientific reports, 9(1), 5233. 10.1038/s41598-019-41695-z

optimize(clusterSizeCompare: int = 1000, resolution: float = 3e-06, intersliceCoupling: float = 0.3, *, maxComSize: bool = False) → None

Optimize clusters accross time slices.

This runs the actual clustering and can be very time and memory consuming for large networks. Depending on the obtained cluster results, this method has to be run iteratively with varying resolution parameter. Output is written to file, with filename containing chosen parameters.

The output CSV contains information on which node in which year belongs to which cluster. As a first measure of returned clustering, the method prints the number of clusters found above a threshold defined by clusterSizeCompare. This does not influence the output clustering.

Parameters:: clusterSizeCompare (int) – Threshold for interesting clusters
Returns:: Tuple of output file path and list of found clusters in tuple format (node, year, cluster)
Return type:: tuple

See also

Documentation of time-layer creation routine: Leiden documentation

Generating reports for time clusters

Create reports for found clusters based on metadata and title and abstract texts.

class semanticlayertools.clustering.reports.ClusterReports(infile: str, metadatapath: Path, outpath: Path, *, textcolumn: str = 'title', authorColumnName: str = 'author', affiliationColumnName: str = 'aff', publicationIDcolumn: str = 'nodeID', numberProc: int = 0, minClusterSize: int = 1000, timerange: tuple = (1945, 2005), rerun: bool = False, debug: bool = False)

Generate reporting on time-clusters.

Generate reports to describe the content for all found clusters above a minimal size by collecting metadata for all publications in each cluster, finding the top 20 authors and affiliations of authors involved in the cluster publications, and running basic BERTtopic modelling. For each cluster a report file is written to the output path.

Input CSV filename is used to create the output folder in output path. For each cluster above the limit, a subfolder is created to contain all metadata for the cluster. The metadata files are assumed to be in JSONL format and contain the year in the filename.

Parameters:

infile (str) – Path to input CSV file containing information on nodeid, clusterid, and year
metadatapath (str) – Path to JSONL (JSON line) formated metadata files.
outpath (str) – Path to create output folder in, foldername reflects input filename
textcolumn (str) – The dataframe column of metadata containing textutal for topic modelling (default=title)
numberProc (int) – Number of CPU the routine will use (default = all!)
minClusterSize (int) – The minimal cluster size, above which clusters are considered (default=1000)
timerange (tuple) – Time range to evalute clusters for (usefull for limiting computation time, default = (1945, 2005))

find_topics(topicDF: list, n_topics: int) → str

Calculate topics in corpus.

Use Bertopic algorithm to calculate topics in corpus file for n_topics topics, returning the 20 most representative words for each topic.

Parameters:

topicDF – The result dataframe of Bertopic.
n_topics (int) – Number of considered topics

Returns:

List of found topics with top occuring words

Return type:

str

fullReport(cluster: int, corpusSizeLimit: int = 100, sampleLimit: int = 10000, *, doSample: bool = True) → str

Generate full cluster report for one cluster.

Parameters:: cluster (int or str) – The cluster number to process
Raises:: ValueError – If input cluster data can not be read.
Returns:: Report text with all gathered informations
Return type:: str

gatherClusterMetadata() → None

Gathering metadata for clusters.

For all files in the metadata path, call _mergeData if the found year in the filename falls in the bounds.

This step needs to be run once, then all cluster metadata is generated and can be reused.

writeReports(corpusSizeLimit: int = 100, *, doSample: bool = True) → None: Generate reports and write to output path.