java—每秒将数据保存到数据库,解决数据库连接丢失或速度慢的问题

7nbnzgx9  于 2021-06-25  发布在  Mysql
关注(0)|答案(2)|浏览(331)

我正在使用spring data jpa和mysql编写java控制台应用程序,并尝试解决以下情况:
应用程序每秒生成一个新对象,该对象应以相同的顺序在同一时刻保存到数据库中。如果db连接将丢失或在对象保存很长时间时调用超时异常,则将即将到来的对象保存到临时缓冲区。恢复连接时,将所有这些累积的对象和新生成的对象(在特定时刻)保存到数据库中。
我的问题是:
当数据库连接丢失时,如何处理将对象保存在临时缓冲区中?
我想我应该在db连接丢失时使用scheduledexecutorservice捕获throwable,然后将特定对象保存到copyonwritearraylist,这是正确的方法吗?
当以前的对象保存调用了连接丢失ot超时异常时,如何停止将新对象保存到db,并在连接打开时恢复保存即将到来的对象的过程?
在保存新生成的对象之前,如何在连接打开时将所有累积的对象保存到db?
更新
我编写的服务使用上述行为运行对象生成:
服务

@Service
public class ReportService implements IReportService {

  @Autowired
  private ReportRepository reportRepository;

  @Override
  public void generateTimestamps() {

    BlockingQueue<Report> queue = new LinkedBlockingQueue<>();
    new Thread(new ReportsProducer(queue)).start();
    new Thread(new ReportsConsumer(queue, reportRepository)).start();
  }

  @Override
  public List<Report> showTimestamps() {
    return reportRepository.findAll();
  }
}

对象生产者:

public class ReportsProducer implements Runnable {

  private final BlockingQueue<Report> reportsQueue;

  ReportsProducer(BlockingQueue<Report> numbersQueue) {
    this.reportsQueue = numbersQueue;
  }

  public void run() {
    try {
      while (true) {
        generateReportEverySecond();
      }
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
    }
  }

  private void generateReportEverySecond() throws InterruptedException {
    Thread.sleep(1000);
    Report report = new Report();
    reportsQueue.put(report);
    System.out.println(Thread.currentThread().getName() + ": Generated report[id='" + report.getId() + "', '" + report
            .getTimestamp() + "']");
  }
}

对象使用者:

public class ReportsConsumer implements Runnable {
  private final BlockingQueue<Report> queue;

  private ReportRepository reportRepository;

  ReportsConsumer(BlockingQueue<Report> queue, ReportRepository reportRepository) {
    this.queue = queue;

//    Not sure i do this correct way
    this.reportRepository = reportRepository;
  }

  public void run() {
    while (true) {
      try {
        if (!queue.isEmpty()) {
          System.out.println("Consumer queue size: " + queue.size());

          Report report = reportRepository.save(queue.peek());
          queue.poll();
          System.out.println(Thread.currentThread().getName() + ": Saved report[id='" + report.getId() + "', '" + report
                  .getTimestamp() + "']");
        }

      } catch (Exception e) {

        //  Mechanism to reconnect to DB every 5 seconds
        try {
          System.out.println("Retry connection to db");
          Thread.sleep(5000);
        } catch (InterruptedException e1) {
          e1.printStackTrace();
        }
      }
    }
  }
}

存储库:

@Repository
public interface ReportRepository extends JpaRepository<Report, Long> {
}

对象:

@Entity
@Table(name = "reports")
public class Report {

  @Id
  @GeneratedValue
  private Long id;

  @Column
  private Timestamp timestamp;

  public Report() {
    this.timestamp = new Timestamp(new Date().getTime());
  }

  public Long getId() {
    return id;
  }

  public void setId(Long id) {
    this.id = id;
  }

  public Timestamp getTimestamp() {
    return timestamp;
  }

  public void setTimestamp(Timestamp timestamp) {
    this.timestamp = timestamp;
  }
}

应用程序属性:

spring.datasource.url=jdbc:mysql://xyz:3306/xyz
spring.datasource.username=xyz
spring.datasource.password=xyz

spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
spring.jpa.properties.hibernate.connection.driver_class=com.mysql.cj.jdbc.Driver
spring.jpa.properties.hibernate.ddl-auto = create-drop

# Below properties don't work actually

spring.jpa.properties.javax.persistence.query.timeout=1

# Reconnect every 5 seconds

spring.datasource.tomcat.test-while-idle=true
spring.datasource.tomcat.time-between-eviction-runs-millis=5000
spring.datasource.tomcat.validation-query=SELECT 1

内部版本.gradle:

version '1.0'

buildscript {
    ext {
        springBootVersion = '1.5.9.RELEASE'
    }
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
    }
}

apply plugin: 'java'
apply plugin: 'org.springframework.boot'

sourceCompatibility = 1.8
targetCompatibility = 1.8

ext {
    mysqlVersion = '6.0.6'
    dbcp2Version = '2.2.0'
    hibernateVersion = '5.2.12.Final'
}

repositories {
    mavenCentral()
}

dependencies {
    compile group: 'mysql', name: 'mysql-connector-java', version: mysqlVersion
    compile group: 'org.hibernate', name: 'hibernate-core', version: hibernateVersion
    compile group: 'org.hibernate', name: 'hibernate-c3p0', version: hibernateVersion
    compile("org.springframework.boot:spring-boot-starter-data-jpa")
    testCompile("org.springframework.boot:spring-boot-starter-test")
    compile group: 'org.assertj', name: 'assertj-core', version: '3.9.0'
}

基于以上代码,我想知道一些时刻:
您建议如何检查数据库连接?
我使用save操作来检查,如果db连接丢失,我只需等待5秒并重复该操作。此外,datasource被配置为在数据库关闭时每5秒建立一次db连接。有没有更正确的方法?
如果db连接处于活动状态,但db当前非常慢或太忙(OVERLOADED),会发生什么情况?
据我所知,我需要设置查询超时。在这种情况下我还应该做什么?

ut6juiuv

ut6juiuv1#

对于前三个问题,您需要 Queue . 如果您使用的是spring框架,那么它确实为jpa使用了一个类,因为 Repository 是一个 Bean 只要配置正确,spring就会为您处理db连接池。
您建议如何检查数据库连接?
您在application.properties中所做的操作会检查连接。我想 5000 毫秒太频繁,可能会减慢系统速度。 360000 可能是个不错的间歇。
如果db连接处于活动状态,但db当前非常慢或太忙(OVERLOADED),会发生什么情况?
配置连接池时,可以设置以下属性:
删除Bandoned- set to true if we want to detect leaked connections 删除BandonedTimeout- the number of seconds from when dataSource.getConnection was called to when we consider it abandoned 日志已放弃- set to true if we should log that a connection was abandoned. If this option is set to true, a stack trace is recorded during the dataSource.getConnection call and is printed when a connection is not returned 参考:为高并发性配置jdbc池

vptzau2j

vptzau2j2#

在我看来,每隔5秒检查一次数据库连接应该不是一个好主意,相反,你会用一些不必要的请求使数据库过载。正如@topdeck提到的,您应该使用 Queue 使用连接池实现(不应通过为addnew对象创建新连接来重载数据库)

相关问题