Article Text

Download PDFPDF

Suicide attempt and intentional self-harm during the earlier phase of the COVID-19 pandemic in Washtenaw County, Michigan
Free
  1. Rachel S Bergmans,
  2. Peter S Larson
  1. Institute for Social Research, University of Michigan, Ann Arbor, Michigan, USA
  1. Correspondence to Dr Rachel S Bergmans, Institute for Social Research, University of Michigan, Ann Arbor, MI 48106, USA; rbergs{at}med.umich.edu

Abstract

Objective Determine the early impact of the COVID-19 pandemic on emergency department (ED) encounters for suicide attempt and intentional self-harm at a regional tertiary academic medical centre in Washtenaw County, Michigan, which is one of the wealthier and more diverse counties in the state.

Methods Interrupted time series analysis of daily ED encounters from October 2015 through October 2020 for suicide attempt and intentional self-harm (subject n=3002; 62% female; 78% Caucasian) using an autoregressive integrated moving average modelling approach.

Results There were 39.9% (95% CI 22.9% to 53.1%) fewer ED encounters for suicide attempt and intentional self-harm during the first 12 weeks of the COVID-19 pandemic (ie, on or after 10 March 2020, when the first cases of COVID-19 were identified in Michigan).

Conclusions Fewer individuals sought emergency care for suicide-related behaviour during the earlier phase of the COVID-19 pandemic than expected when compared to prior years. This suggests initial outbreaks of COVID-19 and state of emergency executive orders did not increase suicide-related behaviour in the short term. More work is needed to determine long-term impacts of the COVID-19 pandemic on suicide-related behaviour and whether there are high-risk groups.

  • suicide
  • time-series
  • social epidemiology
  • clinical epidemiology

Data availability statement

Data came from electronic medical records that are not publicly available.

This article is made freely available for use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.

https://bmj.com/coronavirus/usage

Statistics from Altmetric.com

Introduction

The novel 2019 coronavirus SARS-CoV-2 (COVID-19) pandemic rapidly became a leading cause of death in the United States of America (USA) in 2020,1 and its sweeping economic and social consequences have become a national crisis. Substantial increases in psychological problems due to the COVID-19 pandemic are anticipated, including suicide-related behaviour.2 Suicide represents a substantial public health burden that remains difficult to prevent, even among high-risk patients.3 Fewer than half of those who die by suicide have a known mental health condition, including depression.4 The strongest known risk factor for suicide is having a history of suicide attempt.5 Both suicide and suicide attempts are costly; non-fatal self-inflicted injury in the USA is associated with direct and indirect expenses totalling over $8 billion annually.6 In the wake of the COVID-19 pandemic, some estimated that suicide deaths could increase by as much as 32% in places like Michigan due to a number of factors including delays in help seeking, increased gun purchases, more sleep problems, increased substance use, greater social isolation and economic shocks like unemployment.7

The influence of infectious disease-related public health emergencies like the COVID-19 pandemic on suicide-related behaviour is not well established.8 Infection from coronaviruses like COVID-19 can cause acute neurological symptoms,9 and there have been emerging case reports of suicide or suicide attempt that were attributed to COVID-19 infection and COVID-19-related social impacts.10 11 There is a precedent of elevated suicide-related outcomes following other disaster and crisis events that have economic and social similarities to the COVID-19 pandemic. Increased suicide mortality at the population level was attributed to the USA foreclosure crisis and Great Recession in 2008–2009.12 Following Hurricane Katrina, people reported more suicide ideation and planning—even when accounting for differential exposure to hurricane-related stressors (eg, displacement, life-threatening experience),13 thus demonstrating how crisis events can permeate throughout communities and have widespread mental health impacts. Conversely, community-wide disaster events are also correlated with a decrease in suicide-related behaviour, particularly in cases where economic problems are not a contributing factor.14–16 Some evidence suggests that timing plays an important role in the link between infectious disease-related public health emergencies and suicidality. One study examined changes in suicide-related behaviour in Taiwan over the course of the severe acute respiratory syndrome (SARS) epidemic.17 Huang and colleagues17 observed that overdose-related suicide attempts were highest during peak SARS infection spread when compared with all other (pre, early, late and post) epidemic stages. Even when excluding mortality due directly to COVID-19 in 2020, the USA has experienced an increase in excess deaths above normal trends.18 Further work is needed to determine whether suicide may be contributing to the recent increase in excess deaths.

Michigan is one of the regions in the USA where COVID-19 infection spread was most severe during the initial outbreak compared with other regions of the country. On 10 March 2020, the Michigan Department of Health and Human Services identified the first two presumptive positive cases of COVID-19 in Michigan.19 The following Monday, Michigan’s governor issued the first of many executive orders aimed at reducing community spread of COVID-19, which closed schools and limited the size of most assemblages in the state to <250.20 By April 2020, Michigan had the third highest number of COVID-19 cases and deaths in the USA.21 The economic impact of the COVID-19 pandemic was also relatively high in Michigan, where unemployment rose to 22.7% in April—the second highest in the USA after Nevada.22

This study aimed to determine the early impact of the COVID-19 pandemic on emergency department (ED) encounters for suicide attempt and intentional self-harm at a regional tertiary academic medical centre in Washtenaw County, Michigan. Washtenaw County is one of the wealthier and more diverse counties in the state. Based on USA Census Bureau estimates for 2019,23 Washtenaw County’s population size was 367 601 persons with a median household income of $69 434, which is in the 95th percentile of Michigan counties. Additionally, 74.1% were Caucasian, 12.3% were Black or African American and 13.6% identified as another race or multiple races. During the COVID-19 pandemic from March 2020 through June 2020, Washtenaw County fell into the top 10 Michigan counties for COVID-19 cases, ranking sixth in June.24 In April 2020, unemployment rose to 14.9% from 2.3% the previous month.25

Prior to the COVID-19 pandemic, suicide-related behaviour in the USA was on the rise.26 This pre-existing trend, as well as seasonality of suicide-related behaviour,27 must be accounted for when considering the mental health impact of the COVID-19 pandemic. Therefore, we analysed the temporal patterns of ED encounters at a regional tertiary academic medical centre in Washtenaw County, Michigan; comparing patterns for suicide attempt and intentional self-harm before and after the onset of the COVID-19 pandemic. This quasiexperimental approach tests our hypothesis that the COVID-19 pandemic ‘interrupted’ the time series of ED encounters for suicide attempt and intentional self-harm such that observed trends would differ from expected trends had the COVID-19 pandemic not occurred.

Methods

Sample

The University of Michigan Health System (UMHS) is a regional tertiary academic medical centre that provides single-institution electronic medical records for children and adults who had an ED encounter. UMHS patient records include information on patient demographics, date of ED encounter and reason for ED encounter (recorded as International Classification of Diseases 10th Edition (ICD-10) codes).28

To investigate whether suicide-related behaviour changed after the onset of the COVID-19 pandemic, we examined total daily counts of recorded ED encounters for suicide attempt and intentional self-harm at UMHS facilities. The study time period was 1 October 2015 through 31 October 2020. We limited the data set to start on 1 October 2015 because the UMHS transitioned to using codes from ICD-10 from ICD-9 on that day. The end point of the data was limited to the last day records were available at the time of procurement from UMHS.

Measures

COVID-19 pandemic—A binary measure of ‘before’ and ‘after’ the onset of the COVID-19 pandemic in Michigan was created to examine the impact of the COVID-19 pandemic on daily counts of UMHS ED encounters for suicide attempt and intentional self-harm. We defined the start of the COVID-19 pandemic as the day when the first confirmed cases of COVID-19 were identified in Michigan (10 March 2020).19 Thus, we created a binary variable where days from Sunday, 1 October 2015 through Monday, 9 March 2020 were considered the time period before the COVID-19 pandemic, and we considered the days from Tuesday, 10 March 2020 through Saturday, 31 October 2020 the earlier phase of the COVID-19 pandemic.

Suicide attempt and intentional self-harmonline supplemental table 1 presents the ICD-10 codes used in UMHS to indicate ED encounters for suicide attempt and intentional self-harm. ED encounters for late effects of self-inflected injury (eg, X83.0XXS) were not included. In the time series analyses described below, daily totals of ED encounters were natural log transformed in order to calculate per cent change using effect estimates and due to the skewed nature of the data. A number of days had zero encounters for suicide-related behaviour (n=28). To address zero truncation prior to log transformation, we added one encounter to every day in the time series.

Supplemental material

Approach

Analyses were conducted using R (ver 4.03). First, we conducted a descriptive analysis, tabulating characteristics of all individuals included in the data. We determined sample characteristics by study time period using demographic information (age, gender, race) and other details (eg, number of encounters per person) extracted from the medical record. To visually assess trends over time, we plotted the total number of daily ED encounters for suicide attempt and intentional self-harm including a loess interpolation line for the time periods before and after the onset of the COVID-19 pandemic. Preliminary analyses of data trends also compared patient encounters in March to October for 2020 with the same months from prior years (2016–2019) to limit the influence of seasonal variation. For this analysis, we excluded 2015 data since they were limited to encounters following October and thus did not overlap with the time window of interest.

Next, we used interrupted time series analysis to determine the short-term impact of the COVID-19 pandemic on suicide attempt and intentional self-harm. Specifically, we used an autoregressive integrated moving average (ARIMA) modelling approach. ARIMA is a class of time series analysis that forecasts time series data while accounting for trends within past versions of itself.29 This approach involved using the baseline time series (ie, UMHS encounters from 1 October 2015 to 9 March 2020) to test for (1) non-stationarity using the augmented Dickey-Fuller test and visual inspection; (2) white noise (ie, the time series is purely random); and (3) autocorrelation using the partial autocorrelation function plot and Durbin-Watson statistic. Once non-stationarity and autocorrelation were accounted for as needed, we assessed model fit using the Akaike information criterion estimator and ensuring that the residuals were white noise. An automated process of model selection (using auto.arima from the Automatic Time Series Forecasting package in R)30 identified optimal ARIMA terms. Next, we compared the ARIMA model forecast with observed daily encounters following the onset of the COVID-19 pandemic to test the step effect (before COVID-19=0, after COVID-19 onset=1) and the ramp effect (ie, day-over-day change in encounters).

As a sensitivity analysis, we conducted a change point analysis. A change point is a point in time where the statistical properties, such as the mean value, differ before and after this time.31 Change point analysis is commonly used to determine whether exogenous factors or events impact a time series.32 While we defined the onset of the COVID-19 pandemic as 10 March 2020 a priori, this sensitivity analysis uses the data to determine whether there is statistical evidence to suggest that a particular date represents a change in the mean number of ED encounters.

As an exploratory analysis, we examined whether patient demographics for suicide attempt and intentional self-harm changed in response to the COVID-19 pandemic. Univariate logistic regression tested associations of patient demographic factors (ie, age, gender, race, marital status and number of ED encounters for suicide-related behaviour) with the binary measure for after versus before the COVID-19 pandemic.

Results

In total, there were 3002 individuals in our data (62.0% female; 78.0% Caucasian; mean age=26.4). The median number of ED encounters for suicide attempt or intentional self-harm per person was 2.2 (SD=2.4); 40.1% of patients had a single ED encounter during our study period, 39.9% had two ED encounters and 20.0% had more than two ED encounters. See table 1.

Table 1

Characteristics of emergency department (ED) patients for suicide attempt and intentional self-harm, UMHS 2015–2020

Figure 1 presents the time series for UMHS encounters for the study period. There were 10 753 total encounters for the study period, 1438 encounters occurred after the COVID-19 pandemic onset. Visual inspection suggests that the daily number of ED encounters began to decline overall following 10 March 2020.

Figure 1

Time series plot for suicide attempt and intentional self-harm emergency department (ED) encounters, 1 October 2015 to 31 October 2020. Loess interpolation curves for time periods before and after the COVID-19 pandemic illustrate trend. Red vertical line represents 10 March 2020, when the first cases of COVID-19 were identified in Michigan.

The median number of daily ED encounters was lower following the onset of the COVID-19 pandemic than before, declining from eight ED encounters before the onset of the COVID-19 pandemic to four in the period after (online supplemental figure 1). Median number of ED encounters was also lower for 2020 than all other years when restricting the data to March to October for each year (online supplemental figure 2). Using this limited data set and using 2020 as the reference year, a negative binomial model for the number of ED encounters indicated that the incidence rate ratio was 1.38 (95% CI 1.26 to 1.73) times higher in 2016 with similar findings for years 2017–2019 (online supplemental table 2).

While the baseline time series for patient encounters did not have a seasonal component, non-stationarity was confirmed using the augmented Dickey-Fuller test. First-degree differencing of the baseline time series achieved stationarity. The residuals of an ARIMA(1,1,2) model of the full series indicated normality and thus relieved us of having to aggregate the data to a coarser time division (online supplemental figure 3).

Table 2 presents the maximum likelihood estimates for the step effect and ramp effect models. An ARIMA(2,1,2) was produced to test the step effect. There were 39.9% (95% CI 22.9% to 53.1%) fewer UMHS encounters than expected for suicide attempt and intentional self-harm following the COVID-19 pandemic onset in Michigan. An ARIMA(2,1,2) also tested the ramp effect. Day over day, there was a 0.3% (95% CI 0.1% to 0.5%) decrease in UMHS encounters for suicide attempt and intentional self-harm following the COVID-19 pandemic onset in Michigan.

Table 2

Maximum likelihood estimation for ARIMA testing the short-term effects of the COVID-19 pandemic on emergency department encounters* for suicide attempt and intentional self-harm, UMHS 2015–2020†

Results from the change point analyses indicate that the average number of ED encounters remained consistent until 17 March 2020, with an average of 8.6 ED encounters for suicide-related behaviour per day. After which, the average number of ED encounters dropped to 5.5 per day (online supplemental figure 4).

Table 3 presents the results from logistic regression analyses that tested whether patient demographics changed before versus after the COVID-19 pandemic.

Table 3

Patient demographics for emergency department (ED) encounters after versus before the COVID-19 pandemic, UMHS 2015–2020

Following identification of the first COVID-19 cases in Michigan (ie, 10 March 2020), the proportion of patient encounters for suicide-related behaviour increased among males versus females (OR 1.16; 95% CI 1.04 to 1.31); those aged 18–65 years versus those <18 years (OR 1.41; 95% CI 1.25 to 1.58), and among those with a history of three or more encounters for suicide attempt or intentional self-harm (OR 1.18; 95% CI 1.02 to 1.37). The proportion of patient encounters decreased among Asians versus Caucasians (OR 0.64; 95% CI 0.43 to 0.91), and among married versus unmarried patients (OR 0.56; 95% CI 0.42 to 0.73).

Discussion

We used electronic medical records for ED encounters from a regional tertiary academic medical centre to determine the short-term impact of the COVID-19 pandemic on suicide-related behaviour in Washtenaw County, Michigan. Washtenaw County is one of the wealthier and more diverse counties in Michigan. Our results indicated that there were fewer ED encounters for suicide-related behaviour during the first 7 months of the COVID-19 pandemic than expected when compared to prior years. This suggests that communities like Washtenaw County experienced a decrease in suicide-related behavior following initial outbreak peaks and gubernatorial executive orders for business closures and physical distancing.

Similar to our findings, decreases in suicide mortality have been reported in association with the COVID-19 pandemic elsewhere. In Japan, suicide mortality was 20% lower in April 2020 than April 2019.16 Other preliminary evidence from Japan indicates a similar trend.33 The first case of COVID-19 appeared in Japan on 16 January.34 When cases began to increase rapidly in April, the government declared a state of emergency that encouraged working from home, social distancing practices and closures of some businesses.35 Experts from Japan in suicide prevention hypothesised that the observed decrease could be attributed to fewer social stressors from academic settings and employment as people refrained from leaving their households for non-essential purposes.16 This could also be the case in Michigan, particularly in Washtenaw County, where median household income is higher than the state as a whole and could have provided a buffer against initial financial shocks. Additionally, legislative action on behalf of the USA federal government may have attenuated the mental health impact of employment layoffs and furloughs. On 27 March 2020, the USA Congress passed the Coronavirus Aid, Relief and Economic Security Act, which provided $2 trillion in economic stimulus funds including expansion of unemployment insurance benefits through 31 July 2020.36

Our findings are consistent with Durkheim’s theory of suicide, which emphasises the importance of social integration—the degree of shared values, beliefs and norms between individuals and their community—in preventing suicide.14 Durkheim posits that community-wide disasters like war or the COVID-19 pandemic can lower suicide rates because these events cause individuals to think less of themselves as individuals and more of themselves as part of a community; thus redirecting thoughts from personal well-being to the well-being of one’s community as a whole. This theory has been applied to other evidence of decreases in suicide-related behaviour following national disasters and crises including after the 11 September 2001 terrorist attacks in the USA,15 and the 2011 Fukushima Daiichi nuclear disaster.37

Results also suggest that ED encounters for suicide-related behaviour did not decrease until 17 March 2020, whereas the first cases of COVID-19 were identified a week earlier in Michigan on 10 March. This suggests that trends in suicide-related behaviour were not affected until COVID-19 reached Michigan even though the first identification of COVID-19 in the USA occurred on 20 January 2020 in Washington.38 Additionally, this finding may support the importance of social and economic factors in determining suicide-related outcomes following crisis-like events. For example, people may not have considered the COVID-19 pandemic to personally impact them until the first executive order took effect on 16 March, which closed schools and limited the size of social gatherings in communities state-wide.20

Finally, our exploratory findings suggest that the decrease in suicide-related behaviour did not affect all subpopulations the same. Following the COVID-19 pandemic, the decline in ED encounters was lower among males, Caucasians, patients who were unmarried and those with a greater history of suicide-related ED encounters when compared to other groups. These results are consistent with suicide risk factors. In the USA, suicide mortality is higher among men, Caucasians, those who are unmarried and those with a history of suicide attempt.39 We also observed that the proportion of ED encounters for suicide-related behaviour increased among those aged 18–65 relative to those <18 following the onset of the COVID-19 pandemic. Typically, rates of suicide are higher among youth than adults.39 The closure of schools due to COVID-19 may have decreased exposure to psychological stressors in academic settings. Additionally, adolescents may have had fewer opportunities for suicide-related behaviour or decreased access to means when parents transitioned to remote work in the home due to the COVID-19 pandemic. These findings may inform hypotheses in future work on suicide following crisis-like events. It is possible that proactively reaching out to known patients in mental health services with a history of suicide-related behaviour could further reduce suicide risk in this group immediately following a crisis-like event.

Strengths and limitations

Using 6 years of data and a quasiexperimental approach, we determined the change in suicide attempt and intentional self-harm during the first 7 months of the COVID-19 pandemic in a wealthy and diverse community. Data came from ED encounters for suicide-related behaviour and we excluded encounters for late effects of self-inflicted injury. A strength of these data compared with suicide mortality records is the lack of lag time between when a patient is treated and when their encounter is entered into the medical record. While ED encounters for suicide-related behaviour may not capture every instance of suicide attempt within a community, the time series design provides a robust measure of change in our data since it uses a past version of itself as the control. This inherently accounts for underlying trends, including access to mental health care. Within this time period, Michigan ranked third for number of COVID-19 cases in the USA. Our findings indicate that the earlier phase of the COVID-19 pandemic was associated with a decrease in suicide-related behaviour.

However, our study has limitations to be addressed in future research. First, it is possible that suicide attempts had a higher fatality rate during this time period such that individuals died before they could receive emergency care. Second, since severity of suicide-related behaviour was not provided within the medical record, the decrease in ED encounters may be at least partially attributable to changes in help-seeking behaviour. To explore this further, we created a three-level variable for number of encounters within our data (ie, 1, 2 or >2) and compared how the distribution of this variable differed before and after COVID-19 onset based on the assumption that those with more frequent encounters may be less likely to have life-threatening injuries. We also compared how the distribution of male versus female ED encounters changed after COVID-19 onset since men who attempt suicide tend to have a more serious intent of death than women who attempt suicide.40 ED encounters decreased for all groups. Before COVID-19 onset, those with three or more ED encounters represented 41.8% of all encounters; this increased to 46.4% after COVID-19 onset. Before COVID-19 onset, first-time ED encounters made up 23.4% of all encounters, this decreased to 22.0% after COVID-19 onset. The proportion of ED encounters among those with three or more encounters increased relative to those with a single encounter (OR 1.18; 95% CI 1.02 to 1.37). This indicates that ED encounters decreased less among those with three or more ED encounters when compared to those with a single encounter. However, first-time visits represented over 20% of the ED encounters following onset of the COVID-19 pandemic. The proportion of male ED encounters increased following the onset of the COVID-19 pandemic when compared with ED encounters among women (OR 1.16; 95% CI 1.04 to 1.31). See table 3. This indicates that there was a decrease in both help-seeking and public health need for suicide-related behaviour, but further study is warranted. Third, due to small sample size we could not quantify susceptibility by social or economic status. It is likely that the impact of the COVID-19 pandemic was not equitable across communities and that certain groups were more vulnerable. Fourth, exploratory analyses did not test whether there were changes in method of suicide attempt or intentional self-harm after versus before the COVID-19 pandemic because method was not specified in the medical record for 25.2% of encounters. Finally, we drew data from the earlier phase of the COVID-19 pandemic and it is possible that increases in suicide-related behaviour will emerge over time.

Conclusions

In our study, fewer individuals sought emergency care for suicide-related behaviour during the earlier phase of the COVID-19 pandemic than expected when compared to prior years. This suggests that initial outbreaks of COVID-19 and state of emergency executive orders did not increase suicide-related behaviour. Findings are consistent with prior work examining the impact of national disasters on suicide-related outcomes and Durkheim’s theory of suicide and social integration. Since data came from a regional tertiary academic medical centre in Washtenaw County, Michigan, which is one of the wealthier and more diverse counties in the state, future studies should replicate analyses in other communities. Given that suicide attempt is one measure of mental health, additional research is needed to determine the impact of the COVID-19 pandemic on suicide mortality and other psychiatric conditions including anxiety, depression and post-traumatic stress disorder. More work is also needed to determine the long-term impacts of the COVID-19 pandemic on suicide-related behaviour and whether there are high-risk groups.

What is already known on this subject

  • Infectious disease-related public health emergencies like the novel 2019 coronavirus SARS-CoV-2 (COVID-19) pandemic influence a broad range of social, behavioural and economic factors that can impact mental health and suicide risk. However, few studies have examined the impact of the COVID-19 pandemic on suicide-related behaviour.

What this study adds

  • Using emergency department records from October 2015 to October 2020 and interrupted time series analysis, this study determined whether the earlier phase of the COVID-19 pandemic influenced suicide-related behaviour in a wealthy and diverse Michigan community. During this time, Michigan ranked third in the USA for COVID-19 cases. Results indicated that emergency department encounters for suicide attempt and intentional self-harm were 39.9% (95% CI 22.9% to 53.1%) lower than expected. In conclusion, this study suggests that the earlier phase of the COVID-19 pandemic was associated with a decrease in suicide-related behaviour. Findings are consistent with data from other regions and prior work on community-wide disasters.

Data availability statement

Data came from electronic medical records that are not publicly available.

Ethics statements

Ethics approval

Extraction and analysis of data for this study was deemed human subjects research and approved by the University of Michigan Institutional Review Board (protocol number: HUM00183178).

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors RSB conceived the study, extracted and analysed the data, interpreted the study findings and wrote the initial draft of the manuscript. PSL analysed the data, interpreted the study findings, generated figures and tables and assisted in writing the manuscript.

  • Funding RSB was supported by funding from the USA National Institute for Environmental Health Sciences Michigan Center on Lifestage Environmental Exposures and Disease (NIEHS P30ES017885). PSL was supported through a postdoctoral research fellowship from the University of Michigan Institute for Global Change Biology.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.