Skip to content

Handling purging of lingering files before scheduleExecutor start. #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
khushbr merged 3 commits into from
Jul 14, 2021
Merged

Handling purging of lingering files before scheduleExecutor start. #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
256d65d
Moving deleteAllFiles() to inside scheduleExecutor()
khushbr Jul 14, 2021
28ea9d7
Fixing the Link Checker errors, updating the official documentation
khushbr Jul 14, 2021
466d769
nit: Fixing spotlessJava indentation issue
khushbr Jul 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[![Java CI](https://github.com/opensearch-project/performance-analyzer/workflows/Java%20CI/badge.svg)](https://github.com/opensearch-project/performance-analyzer/actions?query=workflow%3A%22Java+CI%22)
[![CD](https://github.com/opensearch-project/performance-analyzer/workflows/CD/badge.svg)](https://github.com/opensearch-project/performance-analyzer/actions?query=workflow%3ACD)
[![codecov](https://codecov.io/gh/opensearch-project/performance-analyzer/branch/main/graph/badge.svg)](https://codecov.io/gh/opensearch-project/performance-analyzer)
[![Documentation](https://img.shields.io/badge/api-reference-blue.svg)](https://docs-beta.opensearch.org/monitoring-plugins/pa/api/)
[![Documentation](https://img.shields.io/badge/api-reference-blue.svg)](https://opensearch.org/docs/monitoring-plugins/pa/api/)
[![Chat](https://img.shields.io/badge/chat-on%20forums-blue)](https://discuss.opendistrocommunity.dev/c/performance-analyzer/)
![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success)

Expand Down Expand Up @@ -96,7 +96,7 @@ See the [design doc](https://github.com/opensearch-project/performance-analyzer-

## Documentation

Please refer to the [technical documentation](https://docs-beta.opensearch.org/monitoring-plugins/pa/index/) for detailed information on installing and configuring Performance Analyzer.
Please refer to the [technical documentation](https://opensearch.org/docs/monitoring-plugins/pa/index/) for detailed information on installing and configuring Performance Analyzer.

## Contributing

Expand Down
2 changes: 1 addition & 1 deletion build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -486,7 +486,7 @@ afterEvaluate {
url 'https://opensearch.org/downloads.html'
summary '''
Performance Analyzer plugin for OpenSearch.
Reference documentation can be found at https://docs-beta.opensearch.org/.
Reference documentation can be found at https://opensearch.org/docs/.
'''.stripIndent().replace('\n', ' ').trim()
}

Expand Down
35 changes: 18 additions & 17 deletions src/main/java/org/opensearch/performanceanalyzer/writer/EventLogQueueProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,14 @@ public EventLogQueueProcessor(
}

public void scheduleExecutor() {
// Cleanup any lingering files from previous plugin run.
try {
eventLogFileHandler.deleteAllFiles();
} catch (Exception ex) {
LOG.error("Unable to cleanup lingering files from previous plugin run.", ex);
}
lastCleanupTimeBucket = PerformanceAnalyzerMetrics.getTimeInterval(System.currentTimeMillis());

ScheduledFuture<?> futureHandle =
writerExecutor.scheduleAtFixedRate(
this::purgeQueueAndPersist,
Expand Down Expand Up @@ -179,23 +187,16 @@ public void purgeQueueAndPersist() {
}

private void cleanup() {
long currentTimeMillis = System.currentTimeMillis();
if (lastCleanupTimeBucket != 0) {
// Delete Event log files belonging to time bucket older than past filesCleanupPeriod(defaults to 60s)
long currCleanupTimeBucket = PerformanceAnalyzerMetrics.getTimeInterval(currentTimeMillis);
if (currCleanupTimeBucket - lastCleanupTimeBucket > filesCleanupPeriodicityMillis) {
// Get list of files(time buckets) for purging, considered range : [lastCleanupTimeBucket, currCleanupTimeBucket)
List<String> filesForCleanup = LongStream.range(lastCleanupTimeBucket, currCleanupTimeBucket)
.filter(timeMillis -> timeMillis % MetricsConfiguration.SAMPLING_INTERVAL == 0)
.mapToObj(String::valueOf)
.collect(Collectors.toList());
eventLogFileHandler.deleteFiles(Collections.unmodifiableList(filesForCleanup));
lastCleanupTimeBucket = currCleanupTimeBucket;
}
} else {
// First purge since the start-up, cleanup any lingering files.
eventLogFileHandler.deleteAllFiles();
lastCleanupTimeBucket = PerformanceAnalyzerMetrics.getTimeInterval(currentTimeMillis);
// Delete Event log files belonging to time bucket older than past filesCleanupPeriod(defaults to 60s)
long currCleanupTimeBucket = PerformanceAnalyzerMetrics.getTimeInterval(System.currentTimeMillis());
if (currCleanupTimeBucket - lastCleanupTimeBucket > filesCleanupPeriodicityMillis) {
// Get list of files(time buckets) for purging, considered range : [lastCleanupTimeBucket, currCleanupTimeBucket)
List<String> filesForCleanup = LongStream.range(lastCleanupTimeBucket, currCleanupTimeBucket)
.filter(timeMillis -> timeMillis % MetricsConfiguration.SAMPLING_INTERVAL == 0)
.mapToObj(String::valueOf)
.collect(Collectors.toList());
eventLogFileHandler.deleteFiles(Collections.unmodifiableList(filesForCleanup));
lastCleanupTimeBucket = currCleanupTimeBucket;
}
}

Expand Down