Here is how you need to handle a returned check or if you are refunding a credit card charge.
1.) Unapply the payment (or if you are doing a partial return on a credit card adjust the entry so that you have a balance left on the payment). (Make notes in the screen to reference what and why you are doing this).
Go to Accounting / Receivable. Find the customer and then double click on the check to open it up and edit it.
Change the applied amount to zero, and hit enter.
Save and Exit.
2.) Right click on the check that now shows a Balance, and select "Make Balance Adjustment".
Enter in the type of adjustment and any notes.
3.) If it is for a NSF check and you want to create a charge back to the customer for fees you would actually create an invoice for this.
The report you will need to run for accounting is the credit/debit adjustment report.